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For: METHOD AND APPARATUS FOR 
RANDOMIZED VERIFICATION OF 
NEURAL NETS 

STATEMENT SUBMITTED WITH REQUEST FOR PRE- APPEAL BRIEF REVIEW 

This statement is being filed with a Request for Pre- Appeal Brief Review and Notice of 
Appeal filed in response to the Final Office Action dated February 1 ? 2008 received for the 
above-referenced application. 

The system of the presently-claimed invention provides for randomized verification of 
components, such as neural networks. As one example in the specification, Applicant discloses 
that a component, such as a neural net, may be provided by a simulation model of an aircraft 
subsystem. The model may be an approximation of the actual physical system and may 
introduce a degree of error or uncertainty. The approximation error between the system and the 
model may rarely be assessed with certainty and may be expressed in probabilistic terms. Thus, 
even if an algorithm is deterministically verified against a system model, there may remain a 
statistical uncertainty regarding validity of such result, which suggests that the use of a 
deterministic approach to verification may not be suitable. (See, for example, page 36, lines 1-21 
and page 44, lines 1-22 of the originally-filed specification.) Applicant's presently-claimed 
invention provides for randomized verification of the accuracy of a component implemented 
from a model based on using test points for a test of the component that are randomly selected. 
Applicant has found that a randomized verification method may be applicable to a much wider 
spectrum of practical problems than previously developed for the deterministic verification 
approach. (See, for example, page 47, line 11 to page 48, line 11 of the originally-filed 
specification.) 
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The rejection of claims 1-36 under 35 U.S.C. 101 for nonstatutory subject matter (and 
also to the extent this rejection is used in a rejection under 35 U.S.C. 112, first paragraph) is 
hereby traversed and reconsideration is respectfully requested. The Office Action concludes that 
determining accuracy of a component is not statutory and states that to be statutory under 35 
U.S.C. 101, a claim must be directed to a practical application having a final result that is useful 
(specific, substantial and credible); concrete (substantially repeatable / non-unpredictable); and 
tangible (real world / non-abstract). Applicant recites that a component is implemented from a 
model, and the component is tested according to a randomized verification technique to verify 
accuracy according to criteria defined in the claim to verify whether the component is an 
accurate implementation of the model. A number of randomly selected samples is defined 
according to the recited features and the accuracy is verified according to a selected criterion. 
Applicant refers, for purposes of example and further explanation, to page 43, line 6 to page 45, 
line 2 of the originally-filed specification. Applicant submits that that the final result of the 
presently-claimed invention, for example, a component implemented from a model that is 
verified as accurate according to the recited criteria, is useful, concrete and tangible. 

The Examiner concludes an invention that "verifies" has no practical application (top of 
page 4 of the Office Action). Applicant submits that this conclusion is demonstrably incorrect. 
There is no exclusion of patentability under 35 U.S.C. 101 simply because a claimed invention is 
directed to a process of verification. Verification of accuracy of a component implemented from 
a model is a practical application. For example, one practical application would be determining 
if a component has not been accurately implemented from a model of the component, in which 
case, for example, the component could be discarded or indicated as needing to be corrected. In 
fact, the Office Action initially appears to suggest that the claimed invention has too many 
applications (see page 3 of the Office Action). Further, on page 9 of the Office Action, the 
Examiner questions whether the claimed invention would be suitable for a model of flight 
characteristics of an aircraft wing, a model of the spread of a disease throughout a country or a 
model of obesity among people of a given region. Applicant submits that if the same method 
steps, as recited by Applicant, are applied to determine whether a component generated from any 
of the above-noted models is accurately implemented from the model, then it is unclear why the 
Examiner considers that the method steps would not recite an invention having a practical 
application if applicable to all of the above-noted models. Applicant directs particular attention 
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to discussion in the specification, as noted above, concerning applicability of the claimed 
invention to an aircraft subsystem, for example. Accordingly, in view of the above, Applicant 
submits that the rejection should be reconsidered and withdrawn. 

The rejection of claims 1, 16-19, and 34-36 under 35 U.S.C. 101 as lacking patentable 
utility and the rejection of claims 1, 16-17, 19 and 34-35 on the same basis are hereby traversed 
and reconsideration is respectfully requested. The Office Action appears to object to the use of a 
confidence value (5) that is between 0 and 1 and to the accuracy level (e) being between 0 and 1, 
and the use of these variables in determining a number of randomly selected samples M in the 

i fn 

equation M > — In — . In the prior Office Action, the Examiner had concluded that the 

s yd) 

equation somehow indicates that "infinity is less than infinity which makes no sense." (see p. 4, 
August 9, 2007 Office Action.) Applicant respectfully submits that the above-noted confidence 
value (8) and the accuracy level (e) for the selection of randomly selected samples in the above- 
noted equation do not yield the conclusion "infinity is less than infinity" so as to be senseless. 

The equation for selecting the number of randomly selected samples M does suggest that 
as the confidence value and accuracy level decrease in accordance with the recited features, the 
number of randomly selected samples required increases, based on the natural log of one divided 
by the confidence value and one over the accuracy level. The equation characterizes an analysis 
that the smaller the confidence value and more precise the level of accuracy desired, the higher 
the number of random samples required, as dictated by the above-noted equation. Applicant 
submits there is no arbitrary cutoff as the confidence value (5) and the accuracy level (e) become 
smaller and smaller with respect to the number of randomly selected samples recited by the 
claimed invention in accordance with the recited bounds. Instead, the relationship between 
confidence value, accuracy level and number of randomly selected samples is accurately 
characterized by the equations, and defines patentable subject matter in conjunction with the 
other recited features. That is, if a very precise level of accuracy with a very narrow confidence 
interval is desired then many random samples will be required to achieve this. This is the 
relationship defined by the equation according to the variables therein. The Examinees analysis 
of the equation appears somewhat flawed. Applicant submits that the recited features, 
specifically in connection with the equation for determining the minimum number of required 
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samples for a desired confidence value and accuracy level, have utility and would be well 
understood by one of ordinary skill in the art. Accordingly, Applicant respectfully requests that 
the rejection be reconsidered and withdrawn. 

The rejection of claims 12 and 30 under 35 U.S.C. 112, first paragraph, as lacking 
enablement is hereby traversed. The Examiner suggests that there is no explanation for receiving 
a value of zero for p-est, an estimate of p based on a random sample, p, representing a 
probability that a randomly selected data value is in accordance with a selected criterion, F or 
steps relating to receiving 5 or steps relating to receiving 8. Applicant traverses this conclusion. 
Applicant notes that 5 is a confidence value and 8 is an accuracy level, as noted above. 
Applicant also notes that the discussion of p-est begins on page 26, line 13 (p-est being the same 
as p , an estimate of p based on a random sample). As described, p-est = 0 means that an 
observed number of error values exceeding prescribed limits is zero. The description of 
Equations 1 1-18 in the specification detail the result of p-est = 0 and for given values of 5 and 8 
that allow the determining of the equation for the number of random samples required (M). The 
steps of receiving the values for p-est, 5 and 8 in the context of determining the number of 
randomly selected data values (M) seems clear and would be understood by one of ordinary skill 
in the art. Accordingly, Applicant respectfully requests that this rejection should be reconsidered 
and withdrawn. Further, to the extent a similar rejection as that set forth above and that was 
previously applied to claims 1 and 19 is maintained, Applicant submits that the rejection of these 
claims should also be withdrawn. 

The following rejections appeared in the First Office Action dated August 9, 2007, but 
were not included in the Final Office Action dated February 1, 2008. It is unclear to Applicant 
whether or not these rejections have been maintained. To the extent these rejections have been 
maintained, Applicant traverses them for the reasons set forth below. 

The rejection of claims 1-4, 8, 12, 19-22, 26 and 30 under 35 U.S.C. 101 as lacking 
patentable utility because of the use of a selected criterion F is hereby traversed and 
reconsideration is respectfully requested. Applicant's selected criterion F is described in the 
specification (see, for example, beginning page 16, line 9 of the originally-filed specification) 
and recited in connection with a step of a method for determining if each of the randomly 
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selected samples is not in accordance with said selected criterion, F. Applicant discloses 
examples of selected criterion F in the specification and as recited in the dependent claims. For 
example, dependent claim 4 recites that the selected criterion F is that e(x) evaluates to a value 
that exceeds predetermined bounds. Applicant submits that a step of determining if each of 
randomly selected samples is not in accordance with the selected criterion F is fully described 
and enabled by the specification and would be fully understood by one of ordinary skill in the 
art. Accordingly, Applicant respectfully requests that the rejection be reconsidered and 
withdrawn. 

The rejection of claims 4 and 22 under 35 U.S.C. 112, first paragraph, as lacking a 
definition in the specification of the functions e(x), f(x) and (p(x) is hereby traversed. As noted 
beginning on page 13, line 20, f(x) is a neural net trained to replace a known function <p(x) and, 
as noted beginning at page 16, line 1, e(x) is an error function defined by f(x) - (p(x). As defined 
in the claims, x is one of said points corresponding to one or more neural network inputs, where 
(for each point x, evaluated at f(x) and (p(x)), f(x) is a neural network output for a corresponding 
one of said points, and q>(x) is an expected output for a corresponding one of said points. 
Accordingly, an error function determination of e(x) is based on evaluating, for each point x, the 
neural net function f(x) and known function (p(x) that the neural net function is designed to 
replace, with the difference therebetween being the error function e(x). Applicant submits that 
the claimed terms are fully described and enabled by the specification and that one of ordinary 
skill in the art would understand and appreciate the claimed features. The term cp(x) is used for 
a known function, the term f(x) is used for a neural network function trained to model the known 
function, and the term e(x) is used for an error function for the difference between the known 
function and the neural net function. Accordingly, these terms are fully defined. To the extent 
the Examiner is suggesting that specific functions need to be identified, Applicant traverses this 
conclusion, noting that the features of a known function, a neural net function trained to model 
the known function and an error function for the difference therebetween are sufficiently defined 
in the specification and claims. Applicant has used the terms cp(x) for the known function, f(x) 
for the neural net function and e(x) for the difference between the functions. Specific functional 
definitions beyond these definitions are not required, rather, these terms have been adequately 
defined as noted above. Accordingly, Applicant respectfully requests that this rejection be 
reconsidered and withdrawn. 
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